Business Problem:¶
A group of customers were given an offer in person that they can get a loan at discounted rate and processing fee will be waived off. A pilot campaign was conducted to get response from customers whether they are interested in taking out a loan or not. Response was recorded and data was collected.
Need to Do¶
- [x] Build a model to predict whether customers will be interested in taking out a loan or not.
- [x] Identifying features which are most important
- [ ] In case of black box models e.g. Random forest use SHAP, LIME to figure out features affecting the target variable
- [x] Try Unsupervised clustering models
- [ ] Generate synthetic data for model.
- [x] Approaching a customer has costs involved with it, hence find the profitable segments so that more customized marketing can be done.
- [ ] Need to write inferences what is going on
- [x] Bucketing Age and SCR
Variables involved: Customer_id, Age, Gender, Balance, Occupation, No of Credit transaction, SCR, Holding period
Bucketed
Final visualization
age_d == 1,2,3,4 vs Target, SCR , _Balance
Understanding Variables¶
Holding Period(How long the customer is able to hold the money in his account.. So, if they have some existing expenses like a loan EMI or any other monthly expense which gets deducted, usually the first week of every month, hence it makes the balance in the account lower during initial days of the month itself.Higher the holding period, more stable their money is in the account.)SCRSCR is a score given to a customer for a particular product ( in this case loan ) based on certain parameters, to know whether how likely that customer is to buy that product.. so, higher the score, higher the probability, the customer will buy it.. ###SCR propensity of a customer to respond to a digital marketing
Changes v6:¶
Now All Models measure recall on same testing data
Fixed Sampling mistake
Redefined
print_classification_reportasclassification_reportfor better clarity and ease of useVisualized Decision Trees
Implemented SVC
Implemented KNN which provided great results with default parameters
Changes v7:¶
Fit Random Forest Models
Fit XgBoost Models
Changes v9:¶
Added cross validation
Added ROC plots
In [1]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
from sklearn.model_selection import train_test_split, cross_val_score, cross_validate
from sklearn.tree import DecisionTreeClassifier, plot_tree
from sklearn.metrics import accuracy_score, recall_score, f1_score, precision_score, roc_curve, auc, plot_roc_curve
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import plot_partial_dependence
from imblearn.under_sampling import NearMiss
from imblearn.over_sampling import SMOTE
import xgboost as xgb
In [2]:
#from pandasgui import show
In [3]:
data = pd.read_csv('Model_data.csv')
#show(data)
In [4]:
data.Balance = data.Balance.astype('int32') #Truncating decimals
In [5]:
data.head()
Out[5]:
In [6]:
data.Balance.describe()
Out[6]:
In [7]:
data.shape
Out[7]:
In [8]:
data.info()
Gender and Occupation are categorical varibles stored as object type
EDA
No Strong correlations measured except for mild ones in Holding_period and other variables
In [9]:
sns.heatmap(data.corr(), annot=True, square=True) # No strong correlations seen overall
plt.show()
In [10]:
sns.pairplot(data, hue='Target')
plt.show()
In [11]:
sns.countplot(x = data.Gender)
plt.show()
In [12]:
data.Gender.unique()
Out[12]:
In [13]:
data.Occupation.unique()
Out[13]:
In [14]:
data.Gender.value_counts()
Out[14]:
In [15]:
data.drop(data.Gender[data.Gender== 'O'].index, axis = 0, inplace= True) # Removed 196 rows with `Gender` = 'O'
In [16]:
data.shape
Out[16]:
In [17]:
data.Balance.describe()
Out[17]:
In [18]:
# sns.histplot(data.Age)
In [19]:
sns.countplot(x = data.Occupation)
Out[19]:
In [20]:
sns.countplot(x=data.Target, hue=data.Occupation) ## Self employed are much more likely to take loans
Out[20]:
In [21]:
g = sns.FacetGrid(data, col='Occupation', hue="Gender")
plt.grid(True)
g.map(sns.countplot, "Gender", alpha=1)
g.add_legend()
plt.grid((False))
In [22]:
# sns.histplot(data.No_OF_CR_TXNS)
In [23]:
data.No_OF_CR_TXNS.describe()
Out[23]:
In [24]:
sns.violinplot(x=data.No_OF_CR_TXNS)
plt.grid(True)
In [25]:
# len(data[data.No_OF_CR_TXNS==0].index)
In [26]:
# data.drop(index=data[data.No_OF_CR_TXNS==0].index, axis=0)
In [27]:
# sns.displot(data.SCR, kind = 'kde')
sns.distplot(data.SCR)
plt.show()
In [28]:
data.SCR.describe()
Out[28]:
In [29]:
# sns.histplot(data.Holding_Period)
In [30]:
data3 = data
Create a function for easy report printing¶
In [31]:
# A class for pretty printing
class color:
PURPLE = '\033[95m'
CYAN = '\033[96m'
DARKCYAN = '\033[36m'
BLUE = '\033[94m'
GREEN = '\033[92m'
YELLOW = '\033[93m'
RED = '\033[91m'
BOLD = '\033[1m'
UNDERLINE = '\033[4m'
END = '\033[0m'
# function for validation on test data
def classification_report(y_true, y_prediction, type_of_data='Enter Over/Under/Original sampled', type_of_classifier='ClassifierName'):
"""Print Classification report"""
accuracy = accuracy_score(y_true, y_prediction)
precision = precision_score(y_true, y_prediction)
recall = recall_score(y_true, y_prediction)
f1 = f1_score(y_true, y_prediction)
print('Classification Report on Testing Data:\n'+ color.BOLD + type_of_data, 'data\n'+color.END+color.RED+color.BOLD+type_of_classifier,'Classifier'+color.END+color.END)
print()
print('---------------------------------------')
print(color.BOLD + 'Recall: %s' %recall + color.END)
print('Precision: %s' %precision)
print('F1 score: %s' %f1)
print('Accuracy: %s' %accuracy)
print('---------------------------------------')
print()
# A function for cross-validation report
def cross_val_report(classifier, train_data, train_label, cv=10, scoring=['recall','precision', 'f1','accuracy']):
score = cross_validate(classifier, train_data, train_label, cv=cv, scoring= scoring)
recall = np.mean(score['test_recall'])
precision = np.mean(score['test_precision'])
f1 = np.mean(score['test_f1'])
accuracy= np.mean(score['test_accuracy'])
print('Cross Validation Report')
print(color.BOLD + 'Recall: %s' %recall + color.END)
print('Precision: %s' %precision)
print('F1: %s' %f1)
print('Accuracy: %s' %accuracy)
print()
print("*Mean values presented")
print('---------------------------------------')
Create the first set of training and test data on imbalanced data
In [32]:
df = pd.get_dummies(data, columns=['Gender','Occupation'], drop_first = True)
df.head()
Out[32]:
Creating a model with Original Unbalanced data and measuring metrics
In [33]:
X_original = df.iloc[:,1:]
y_original = df.iloc[:,0]
In [34]:
X_original
Out[34]:
In [35]:
X_train_orig, X_test_orig, y_train_orig, y_test_orig = train_test_split(X_original,y_original, shuffle = ['True'], stratify=y_original)
In [36]:
clf = DecisionTreeClassifier(max_depth = 5)
clf.fit(X_train_orig, y_train_orig)
y_prediction_orig = clf.predict(X_test_orig)
classification_report(y_test_orig, y_prediction_orig, 'Original', 'Decision Tree')
plot_roc_curve(clf, X_test_orig, y_test_orig)
plt.show()
# cross_val_report(clf, y_test_orig,y_under_prediction.reshape(1,-1))
In [37]:
from sklearn.metrics import confusion_matrix
In [38]:
confusion_matrix(y_test_orig, y_prediction_orig)
Out[38]:
In [39]:
# fig = plt.figure(figsize=(50,20))
# _ = plot_tree(clf,
# feature_names=list(X_original.columns),
# class_names=['0','1'],
# filled=True, fontsize=10)
Create undersampled data and fit a model
In [40]:
X_under_train, y_under_train = NearMiss().fit_resample(X_train_orig, y_train_orig)
In [41]:
data[data.Target==1].shape
Out[41]:
In [42]:
X_under_train.shape, y_under_train.shape
Out[42]:
In [43]:
clf_under_sampled = DecisionTreeClassifier(max_depth = 5)
clf_under_sampled.fit(X_under_train, y_under_train)
y_under_prediction = clf_under_sampled.predict(X_test_orig)
classification_report(y_test_orig,y_under_prediction, 'Undersampled', 'Decision Tree')
plot_roc_curve(clf_under_sampled, X_test_orig, y_test_orig)
Out[43]:
In [44]:
# cross_val_report(clf_under_sampled, y_test_orig,y_under_prediction)
## crossval here causes unbalanced split
In [45]:
# fig = plt.figure(figsize=(100,100))
# _ = plot_tree(clf_under_sampled,
# feature_names=list(X_original.columns),
# class_names=['0','1'],
# filled=True, fontsize=10)
Model on an oversampled dataset
In [46]:
X_over_train, y_over_train = SMOTE().fit_resample(X_original, y_original)
In [47]:
clf_over_sampled = DecisionTreeClassifier(max_depth = 5)
clf_over_sampled.fit(X_over_train, y_over_train)
y_over_predict = clf_over_sampled.predict(X_test_orig)
classification_report(y_test_orig, y_over_predict, 'Oversampled', 'Decision Tree')
plot_roc_curve(clf_over_sampled, X_test_orig, y_test_orig)
Out[47]:
In [48]:
# fig = plt.figure(figsize=(100,100))
# _ = plot_tree(clf_over_sampled,
# feature_names=list(X_original.columns),
# class_names=['0','1'],
# filled=True, fontsize=10)
In [49]:
print("Original: "+color.BOLD+ "X_original,y_original"+color.END+":: X_train_orig, X_test_orig, y_train_orig, y_test_orig")
print()
print("Undersampled:"+color.BOLD+ " X_under, y_under"+color.END+" :: X_under_train, y_under_train")
print()
print("Oversampled:"+color.BOLD+ " X_over, y_over"+color.END+" :: X_over_train, y_over_train")
The above datasets can be better sampled by adjusting hyper-parameters of NearMiss and SMOTE, or other methods of sampling could be used
SVM Classifiers applied¶
SVC fails to fit on original dataset, possibly because of unbalance
In [50]:
clf_svc0 = SVC()
clf_svc0.fit(X_under_train, y_under_train)
y_predict = clf_svc0.predict(X_test_orig)
classification_report(y_test_orig, y_predict, 'Undersampled', 'SVM')
plot_roc_curve(clf_svc0, X_test_orig, y_test_orig)
cross_val_report(clf_svc0, X_under_train, y_under_train)
In [51]:
# %%time
# # Will take LONG Time for Training
# clf_svc1 = SVC()
# clf_svc1.fit(X_over_train, y_over_train)
# y_predict = clf_svc1.predict(X_test_orig)
# classification_report(y_test_orig, y_predict, 'Oversampled', 'SVM')
# plot_roc_curve(clf_svc1, X_test_orig, y_test_orig)
# cross_val_report(clf_svc1, X_under_train, y_under_train)
In [52]:
clf_KNN0 = KNeighborsClassifier()
clf_KNN0.fit(X_train_orig, y_train_orig)
y_predict= clf_KNN0.predict(X_test_orig)
classification_report(y_test_orig, y_predict, 'Original', 'KNN')
plot_roc_curve(clf_KNN0, X_test_orig, y_test_orig)
cross_val_report(clf_KNN0, X_under_train, y_under_train)
In [53]:
clf_KNN1 = KNeighborsClassifier()
clf_KNN1.fit(X_under_train, y_under_train)
y_predict= clf_KNN1.predict(X_test_orig)
classification_report(y_test_orig, y_predict, 'Undersampled', 'KNN')
plot_roc_curve(clf_KNN1, X_test_orig, y_test_orig)
cross_val_report(clf_KNN1, X_under_train, y_under_train)
Random Forest Classifier Models¶
In [54]:
clf_rf0 = RandomForestClassifier()
clf_rf0.fit(X_train_orig, y_train_orig)
y_predict= clf_rf0.predict(X_test_orig)
classification_report(y_test_orig, y_predict, 'Original', 'Random Forest')
plot_roc_curve(clf_rf0, X_test_orig, y_test_orig)
cross_val_report(clf_rf0, X_under_train, y_under_train)
In [55]:
plot_partial_dependence(clf_rf0, X_test_orig, X_test_orig.columns)
Out[55]:
In [56]:
(pd.Series(clf_rf0.feature_importances_, index=X_test_orig.columns)
.nlargest(4)
.plot(kind='bar'))
Out[56]:
RFE Trial¶
In [57]:
from sklearn.feature_selection import RFE
In [58]:
rfe = RFE(RandomForestClassifier(), n_features_to_select=4)
rfe = rfe.fit(X_over_train, y_over_train)
# summarize the selection of the attributes
print(rfe.support_)
print(rfe.ranking_)
In [59]:
categorical_features_indices = np.where(X_train_orig.dtypes != np.float)[0]
In [60]:
#importing library and building model
from catboost import CatBoostRegressor
model=CatBoostRegressor(iterations=50, depth=3, learning_rate=0.1, loss_function='RMSE')
model.fit(X_train_orig, y_train_orig,cat_features=categorical_features_indices,eval_set=(X_test_orig, y_test_orig),plot=True)
Out[60]:
In [62]:
shap.summary_plot(shap_values, X_test_orig)
In [63]:
help(CatBoostClassifier)
In [64]:
X_train_orig.columns
Out[64]:
In [65]:
clf_rf1 = RandomForestClassifier()
clf_rf1.fit(X_under_train, y_under_train)
y_predict= clf_rf1.predict(X_test_orig)
classification_report(y_test_orig, y_predict, 'Undersampled', 'Random Forest')
plot_roc_curve(clf_rf1, X_test_orig, y_test_orig)
cross_val_report(clf_rf1, X_under_train, y_under_train)
In [66]:
clf_rf2 = RandomForestClassifier()
In [67]:
clf_rf2.fit(X_under_train, y_under_train)
y_predict= clf_rf2.predict(X_test_orig)
classification_report(y_test_orig, y_predict, 'Oversampled', 'Random Forest')
plot_roc_curve(clf_rf2, X_test_orig, y_test_orig)
cross_val_report(clf_rf2, X_under_train, y_under_train)
In [68]:
(pd.Series(clf_rf2.feature_importances_, index=X_test_orig.columns)
.nlargest(4)
.plot(kind='barh'))
Out[68]:
In [69]:
# rfe = RFE(KNeighborsClassifier(), n_features_to_select=4)
# rfe = rfe.fit(X_train_orig, y_train_orig)
# # summarize the selection of the attributes
# print(rfe.saupport_)
# print(rfe.ranking_)
In [70]:
lr0 = LogisticRegression(max_iter=1000)
lr0.fit(X_train_orig, y_train_orig)
y_predict= lr0.predict(X_test_orig)
classification_report(y_test_orig, y_predict, 'Original', 'Logistic Regression')
plot_roc_curve(lr0, X_test_orig, y_test_orig)
cross_val_report(lr0, X_under_train, y_under_train)
In [ ]:
lr1 = LogisticRegression(max_iter=100)
lr1.fit(X_under_train, y_under_train)
y_predict= lr1.predict(X_test_orig)
classification_report(y_test_orig, y_predict, 'Undersampled', 'Logistic Regression')
plot_roc_curve(lr1, X_test_orig, y_test_orig)
cross_val_report(lr1, X_under_train, y_under_train)
In [ ]:
lr2 = LogisticRegression(max_iter=100)
lr2.fit(X_over_train, y_over_train)
y_predict= lr2.predict(X_test_orig)
classification_report(y_test_orig, y_predict, 'Oversampled', 'Logistic Regression')
plot_roc_curve(lr1, X_test_orig, y_test_orig)
cross_val_report(lr1, X_under_train, y_under_train)
In [ ]:
clf_KNN2 = KNeighborsClassifier()
clf_KNN2.fit(X_over_train, y_over_train)
y_predict= clf_KNN2.predict(X_test_orig)
classification_report(y_test_orig, y_predict, 'Oversampled', 'KNN')
plot_roc_curve(clf_KNN2, X_test_orig, y_test_orig)
cross_val_report(clf_KNN2, X_under_train, y_under_train)
Segment the customer with prob >90, 80>prob>90, 70>prob>80, 60>prob>70 and so on
In [ ]:
predict_prob = clf_KNN2.predict_proba(X_test_orig)
predict_prob.shape
# print more significant digits here
In [ ]:
# predict_prob
In [ ]:
predictions = pd.DataFrame(predict_prob, columns=[0, 'PredictionProb'])
In [ ]:
# predictions.head()
In [71]:
predictions_prob = predictions.drop(0, axis=1)
In [72]:
y = pd.DataFrame(y_test_orig, columns = ['Target'])
y = y.reset_index(drop=True)
In [73]:
error = y.Target- predictions_prob.PredictionProb
In [74]:
# This series indicate the error, closer to zero values mean better prediction
# positive mean that my model should have predicted loan taker, but did not
# negative means that model should have predicted NOT a loan taker, but did predict as such
In [75]:
# target - probablity
In [76]:
plt.hist(error)
plt.xlabel('Error in Prediction: Target - Probablity Of Prediction')
plt.ylabel("Count")
plt.show()
PRE-BUCKETING VISUALIZATIONS FOR INFERENCES¶
In [77]:
sns.catplot(x="Occupation", y="SCR", hue = "Target",data=data, kind = "violin", split = True)
plt.show()
In [78]:
sns.relplot(x="Occupation", y="SCR", hue = "Target",data=data, aspect=1.5, kind = "line")
plt.show()
In [79]:
sns.catplot(x="Occupation", y="SCR", hue = "Target",data=data, aspect=2.0, kind = "point")
plt.show()
In [80]:
sns.catplot(x="Occupation", y="SCR", hue = "Target",data=data, kind = "violin", split = True)
plt.show()
The model is good, make few mistakes
In [81]:
data1 = data.copy()
Bucketing for AGE values¶
In [82]:
data1['Ageb'] = pd.qcut(data['Age'], q=4)
In [83]:
data1.head()
Out[83]:
In [84]:
data1.Ageb.value_counts()
Out[84]:
In [85]:
data1.dtypes
Out[85]:
In [86]:
data1.Ageb
Out[86]:
In [87]:
data1.head(5)
Out[87]:
In [88]:
# sns.pairplot(data1, hue='Target')
In [89]:
data1 = pd.get_dummies(data1)
In [90]:
data1.head(2)
Out[90]:
In [91]:
data2 = data1.copy()
In [92]:
data1.columns
Out[92]:
In [93]:
cols = ['Ageb_(30.0, 38.0]', 'Ageb_(20.999, 30.0]','Ageb_(46.0, 55.0]','Ageb_(38.0, 46.0]']
def func1(x):
if x ==1:
return 1
def func2(x):
if x ==1:
return 2
def func3(x):
if x ==1:
return 3
def func4(x):
if x ==1:
return 4
data1['age_d'] = data1['Ageb_(30.0, 38.0]'].apply(func1)
data1['age_d'] = data1['Ageb_(20.999, 30.0]'].apply(func2)
data1['age_d'] = data1['Ageb_(46.0, 55.0]'].apply(func3)
data1['age_d'] = data1['Ageb_(38.0, 46.0]'].apply(func4)
In [94]:
for ind, row in data1.iterrows():
if row['Ageb_(20.999, 30.0]'] ==1:
data1.loc[ind, 'age_d'] = 1
elif row['Ageb_(30.0, 38.0]'] ==1:
data1.loc[ind, 'age_d'] = 2
elif row['Ageb_(38.0, 46.0]'] ==1:
data1.loc[ind, 'age_d'] = 3
elif row['Ageb_(46.0, 55.0]'] ==1:
data1.loc[ind, 'age_d'] = 4
In [95]:
data1.age_d.value_counts()
Out[95]:
In [96]:
data1.age_d = data1.age_d.astype('int32')
In [97]:
data1.head()
Out[97]:
POST-BUCKETING VISUALIZATIONS FOR INFERENCES¶
In [98]:
sns.catplot(x="age_d", y="SCR", hue = "Target",data=data1, kind = "violin", split = True)
plt.show()
In [99]:
data1.columns
Out[99]:
In [100]:
sns.scatterplot(x='age_d', y="SCR", hue = "Target",data=data1,)
plt.show()
In [101]:
sns.catplot(x="age_d", y="SCR", hue = "Target",data=data1, kind = "swarm")
plt.show()
In [102]:
sns.set_style('ticks')
In [103]:
g = sns.relplot(x="SCR", y="age_d", hue = "Target",data=data1, aspect=3, kind = "line")
# g.figure.set_size_inches(18.5, 10.5)
# sns.despine()
BUCKETING SCR & HOLDING PERIODS¶
In [104]:
data1['SCRb'] = pd.qcut(data['SCR'], q=5)
In [105]:
data1.SCRb.value_counts()
Out[105]:
In [106]:
data1['HPB'] = pd.qcut(data['Holding_Period'], q=4)
In [107]:
data1.HPB.value_counts()
Out[107]:
In [108]:
data1.head()
Out[108]:
In [109]:
sns.catplot(x="age_d", y="SCRb", hue = "Target",data=data1, kind = "violin", split = True)
plt.show()
In [110]:
sns.catplot(x="age_d", y="HPB", hue = "Target",data=data1, kind = "violin", split = True)
plt.show()
In [116]:
sns.pairplot(data1, hue = 'Target')
plt.show()
In [ ]:
SHAP¶
In [111]:
import shap
In [112]:
X_train_summary = shap.kmeans(X_train_orig, 10)
In [113]:
explainerKNN = shap.KernelExplainer(clf_KNN0.predict,X_train_summary)
In [114]:
shap_values_KNN_test = explainerKNN.shap_values(X_test_orig)
In [ ]:
shap.initjs()
shap.force_plot(explainerKNN.expected_value, shap_values_KNN_test[:1000,:], X_test_orig.iloc[:1000,:])
LIME¶
In [ ]:
import lime
import lime.lime_tabular
In [ ]:
lgb_params = {
'task': 'train',
'boosting_type': 'goss',
'objective': 'binary',
'metric':'binary_logloss',
'metric': {'l2', 'auc'},
'num_leaves': 50,
'learning_rate': 0.1,
'feature_fraction': 0.8,
'bagging_fraction': 0.8,
'verbose': None,
'num_iteration':100,
'num_threads':7,
'max_depth':12,
'min_data_in_leaf':100,
'alpha':0.5}
In [ ]:
import warnings
from lime import submodular_pick
import lightgbm as lgb
lgb_train = lgb.Dataset(X_train_orig, y_train_orig)
lgb_eval = lgb.Dataset(X_test_orig, y_test_orig)
model = lgb.train(lgb_params,lgb_train,num_boost_round=20,valid_sets=lgb_eval,early_stopping_rounds=5)
lime.lime_tabular.LimeTabularExplainer(data[model.feature_name()].astype(int).values,
mode='classification',training_labels=data3['Target'],feature_names=model.feature_name())
# Remember to convert the dataframe to matrix values
# SP-LIME returns exaplanations on a sample set to provide a non redundant global decision boundary of original model
sp_obj = submodular_pick.SubmodularPick(explainer,data3[model.feature_name()].values, \
prob, num_features=5,num_exps_desired=10)
[exp.as_pyplot_figure(label=1) for exp in sp_obj.sp_explanations]
In [119]:
from catboost import CatBoostClassifier
model = CatBoostClassifier(iterations=1500, learning_rate=0.01, l2_leaf_reg=3.5, depth=5, rsm=0.98,
loss_function= 'Logloss', eval_metric='AUC',use_best_model=True,random_seed=42)
In [122]:
cate_features_index = np.where(data.dtypes != float)[0]
In [123]:
model.fit(X_train_orig,y_train_orig,cat_features=cate_features_index,eval_set=(X_test_orig,y_test_orig))
Out[123]:
In [131]:
test = pd.read_csv("F:\\IBPA_IIM_FinalProject\\Test_targets.csv")
pred = model.predict_proba(test.iloc[:,0])
In [ ]: